System for providing synchronized sharing of augmented reality content in real time across multiple devices

ABSTRACT

The invention generally relates to managing and sharing augmented reality (AR) content, and, more specifically, to an AR platform providing synchronized sharing of AR content in real time across multiple AR devices.

FIELD OF THE INVENTION

The invention relates to augmented reality platforms, and, moreparticular, to a system for providing synchronized sharing of augmentedreality content in real time across multiple augmented reality-capabledevices within a given physical environment or space.

BACKGROUND

Augmented reality (AR) is a live view of a physical, real-worldenvironment in which elements are “augmented” by computer-generatedperceptual information. Unlike virtual reality, which creates a totallyartificial environment, augmented reality uses the existing environmentand overlays new information on top of it. The overlaid information maybe constructive (i.e. additive to the natural environment) ordestructive (i.e. masking of the natural environment). In particular,the overlaid, computer-generated information is spatially registeredwith the physical world such that the overlaid information may beperceived as an immersive aspect of the real environment. As such,augmented reality is intended to alter a user's current perception of areal-world environment, as opposed to virtual reality that replaces thereal-world environment with a simulated one.

One of the benefits of augmented reality is that it allows components ofthe digital world to be brought into a person's perception of the realworld through the integration of immersive sensations that are perceivedas natural parts of an environment. For example, augmented realitysystems may enhance a person's conception of reality through a varietyof sensory modalities, such as visual, auditory, haptic, and olfactory.Most augmented reality systems provide a wearable device, generally inthe form of a headset to be worn by the person which includes a video orgraphic display through which augmented views of the real-worldenvironment are presented to the wearer. Some augmented reality systemsallow for a person to use a personal computing device equipped withappropriate camera hardware and a display, such as a smartphone ortablet.

While current systems may be able to provide a person with some form ofaugmented reality experience, current systems have drawbacks. Mostnotably, current augmented reality systems, as well as virtual realitysystems for that matter, generally provide a personal-centricexperience, which is focused solely on the immediate user's movement andinteraction with the augmented reality content. As such, use of currentaugmented reality systems can often be an isolating, individualexperience.

SUMMARY

The present invention is directed to a system including an augmentedreality (AR) platform providing synchronized sharing of AR content inreal time and across multiple AR-capable devices within a controlled,physical environment or space. In particular, the system of the presentinvention includes a mesh network of technologies integrated with oneanother and used to ultimately establish alignment of digital content,including rendering thereof, against a physical environment or space.Such a system allows for multiple users to experience the same ARcontent rendering in real time and within a live, physical environmentor space, wherein such rendering of AR content is adapted to each user'spoint of view.

More specifically, the system includes the use of a physical, real-worldenvironment, preferably a controlled space (i.e., a room or at leastpartially enclosed space) in which the AR content is to be presented tomultiple users (via each user's AR-capable device). The use of acontrolled space allows for the system of the present invention toprovide a persistently shared experience dedicated to the specificspace. In some embodiments, the environment may include multiplecontrolled spaces that are part of an overall AR experience to beprovided to the users (i.e., multiple rooms or spaces at a particularvenue, such as multiple spaces representing various exhibits at anAR-based zoo).

For any given controlled space, a shared point is initially established(also referred to herein as “world origin point” or “world origin”). Theworld origin point is generally defined as a specific position andorientation within the given space, which may be based on coordinatedata (e.g., a coordinate axis system, including an x,y,z position andx,y,z orientation). Establishing a world origin point within thecontrolled space allows for the AR platform to place digital contentrelative to the world origin point for subsequent rendering acrossmultiple AR-capable devices. The controlled space is digitally mapped,such that digital data associated with the controlled space, includingthe world-origin point coordinate data, is stored for subsequentretrieval and use during rendering of AR content. Each participatingAR-capable device (i.e., AR-headset, smartphone, tablet, or othercomputing device that is AR-capable) within the controlled spaceessentially agrees upon the established world origin point, therebyallowing for digital content (e.g., images) to consistently appear inthe same, real-world location in the controlled space for eachindividual device as a result of one or more localization and subsequentre-localization processes for each device, as described in greaterdetail herein.

In addition, the system further relies on image tracking for alignmentpurposes. For example, the physical space can be decorated using imagemarker technology. Use of image marker technology allows for canonicallyestablished images to represent coordinates associated with the worldorigin point. For example, at the start of a given AR session orexperience, devices with image tracking technology can utilize one ormore image trackers (i.e. physical markers) within a given space tolocalize into the space and align the AR session to the world originpoint. The localized coordinates of each image marker along with aunique image marker identifier is stored for each image for subsequentretrieval and use by each device, thereby allowing devices to understandthe space without requiring any individual device setup.

The AR platform further coordinates the world origin point of a givencontrolled space with anchor-based localization to thereby align themultiple devices. In particular, each device may be running ananchor-based software algorithm unique to that device's given platform.Anchors are understood to include generated locations that represent aphysical location of the associated device in the real world and storedas serialized data (e.g., in the form of coordinate data). In someembodiments, the devices may be running respective cloud anchoringsystems. Additionally, some devices may be running respective persistentanchoring systems. For each cloud anchoring system, for example, anchorswill be established for each integrated platform in a similar manner toimage markers. However, in the present system, cloud anchors areestablished using a computer vision-based mesh understanding of thephysical world. As previously described, each device within thecontrolled space essentially agrees upon the established world originpoint, such that each device localizes into the space based, at least inpart, on established anchors for that device (i.e., correlation ofanchor data with world origin point data).

Upon a set of devices localizing into the controlled space using atleast one of the image tracking and cloud anchoring techniques, the ARplatform allows for dynamic, real-time localization across all devicesin the given space. Each device will determine, through a series ofchecks, whether to start generating temporary cloud anchors for moreaccurately sharing an AR experience with new devices that enter thespace. As image tracking can require positioning devices in closeproximity to image markers, temporary cloud anchors provide an advantageof allowing more devices to arbitrarily localize into the space withouthaving a multitude of viewers try to crowd into the same vantage point.

The system of the present invention further accounts for drift. Forexample, devices may be continuously re-localizing into the real worldthrough a series of sensors, which may include an RGB camera, Lidarsensors, inertial measurement unit (IMU), motion sensors, infrared, orother tracking system. Such sensors are all subject to disruption, whichcan interfere with the device's understanding of its position andorientation in the real-world environment. Accordingly, as a result ofsuch disruption, the digital AR content provided may shift from itsoriginally localized world origin, resulting in a phenomenon known asdrift, which can cause digitally placed objects to shift to incorrectlocations as a result.

To counter the effects of drift and to make the system easy to use foreach user, the system of the present invention provides for automaticand repeated localization (i.e., re-localization) for any device. Inparticular, for a given AR experience that may include multiplecontrolled spaces (e.g., multiple exhibits in an AR-based zoo, forexample), multiple locations within the real-world environment may bedesignated as re-localization points, in which any given user'sproximity may be detected via a proximity sensor, such as a near-fieldcommunication-based device. For example, proximity sensors may includeBluetooth Low-Energy (BLE) sensors. Upon being detected, a near-fieldcommunication-based sensor may communicate with the AR platform and/ordevice and subsequently initiate a re-localization process, in which thedevice will automatically attempt to re-localize (requiring no directinput or interaction from the user). Such re-localization points can beplaced throughout a given AR experience at regular intervals that users(i.e., guests or participants) must necessarily pass through and areencouraged to come closer as part of the attraction(s). Accordingly, thesystem of the present invention provides for continuous re-alignment ofthe dynamic world origin point through a combination of the use of thephysical image markers as well as disparate cloud services of eachdevice to maintain the associated coordinates consistently across devicesoftware systems throughout the duration of each AR session/experience.

Accordingly, the system of the present invention addresses the drawbacksof current augmented reality systems by recognizing the potential of howexperiential augmented reality can be when experiencing such contenttogether by many at the same time. The AR platform provides forsynchronized sharing of AR content in real time and across multipleAR-capable devices, thereby allowing multiple users to experience thesame AR content rendering in real time and within a live, physicalenvironment or space, wherein such rendering of AR content is adapted toeach user's point of view. The synchronization of content allows formultiple users within the given space to more naturally interface withthe shared AR content as well as observe an identical combination ofdigital and physical reality, thereby simultaneously experiencing andinteracting with augmented reality environments. The AR platform allowsfor the display of AR content within the same physical location andorientation across multiple AR-capable devices, regardless of thedevices being from identical or different manufactures. By combiningdifferent device types together, the system of the present invention isaccessible by most device owners, providing similar AR experiences toboth the handheld mobile market (i.e., smartphones or tablets) and themore expensive lightweight eyewear market. Additionally, by integratingand leveraging multiple technologies (i.e., image tracking technology,cloud-based anchor systems, local persistent anchoring systems, andre-localization proximity sensors), the system of the present inventionis able to ensure constant re-localization that does not depend solelyon a single technology. Based on the communication capabilities (e.g.,network communications), reliability can be shared across the differentplatforms, thereby improving the overall AR experience for all users.

By providing a truly immersive and shared AR experience, systems of thepresent invention can be particularly beneficial in various industriesthat cater to, or otherwise rely on, multiple guests, participants,patrons, or the like. For example, the system of the present inventionmay be particularly useful in the entertainment industry in which agiven venue provides entertainment to multiple guests at once, such as azoo, theme park, sporting event, or the like. Similarly, the systems ofthe present invention may be useful for educational purposes (i.e.,classroom environment in which the instructor and associated courselesson is provided to multiple students via an AR experience provided oneach student's AR-capable device) or military exercises (i.e., soldierscan train via customized training scenarios provided via an ARexperience, including multi-user combat situations).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating one embodiment of an exemplarysystem for providing synchronized sharing of augmented reality contentacross multiple devices.

FIG. 2 is a block diagram illustrating the augmented reality (AR)platform of FIG. 1 in greater detail.

FIG. 3 is a block diagram illustrating the various databases in greaterdetail.

FIG. 4 is a block diagram illustrating at least one embodiment of acomputing device (i.e., smartphone or tablet) for communicating with theAR platform and for subsequently conveying an AR experience to anassociated user based on communication with at least the AR platform.

FIG. 5 is a block diagram illustrating at least one embodiment of acomputing device (i.e., wearable headset) for communicating with the ARplatform and for subsequently conveying an AR experience to anassociated user based on communication with at least the AR platform.

FIG. 6 shows a perspective view of an exemplary wearable headset of thesystem of the present invention.

FIG. 7 is a block diagram illustrating communication between multipleAR-capable devices and the AR platform for localization andre-localization thereof based on at least one of image trackingtechnology, anchor-based technology, and proximity sensing.

FIG. 8 is an exemplary layout of a venue comprised of multiplecontrolled spaces, each having an established world origin point for agiven attraction, at least one image tracking marker, andre-localization zones.

FIGS. 9A-9F show a continuous flow diagram illustrating a method forinitial localization of one or more AR-capable devices within acontrolled environment or space prior to commencing an AR experience orsession.

FIG. 10 is a block diagram of one embodiment of a method for initiatinga re-localization session of an AR-enabled device.

FIG. 11 is a block diagram of another embodiment of a method forinitiating a re-localization session of an AR-enabled device.

FIGS. 12-20 are images depicting various implementations and uses of thesystem of the present invention.

FIGS. 12-16 are images depicting an AR-based zoo experience for multipleguests within controlled spaces (i.e., specific “exhibits”), in whichsystems of the present invention provide synchronized sharing ofzoo-based AR content (i.e., zoo-related animals) in real time and acrossmultiple AR-capable devices (i.e., wearable headsets and/or personalcomputing devices, such as a smartphone or tablet).

FIG. 17 is an exemplary layout or map of an AR-based zoo experience,illustrating the various “exhibits”.

FIG. 18 is an image depicting an AR-based classroom experience, in whichthe instructor and associated course content is provided to multiplestudents via an AR experience, wherein each student is wearing anAR-capable headset.

FIG. 19 is an image depicting another embodiment of an AR-basedclassroom experience, in which the instructor and associated coursecontent/lesson is provided to multiple students via an AR experience,wherein each student is viewing and interacting with the coursecontent/lesson and instructor via a tablet computing device, furtherillustrating the multiple point of views for each student, adding to therealism and feel. Such an experience is particularly useful for distanceeducation, such as remote learning or the like.

FIG. 20 is an image depicting an AR-based military experience, in whichmultiple soldiers are provided with a military training scenario.

DETAILED DESCRIPTION

The present invention is directed to a system including an augmentedreality (AR) platform providing synchronized sharing of AR content inreal time and across multiple AR-capable devices within a controlled,physical environment or space. In particular, the system of the presentinvention includes a mesh network of technologies integrated with oneanother and used to ultimately establish alignment of digital content,including rendering thereof, against a physical environment or space.Such a system allows for multiple users to experience the same ARcontent rendering in real time and within a live, physical environmentor space, wherein such rendering of AR content is adapted to each user'spoint of view.

The AR platform, for example, is accessible to users via associatedAR-capable computing devices, including certain personal computingdevices (i.e., smartphones and tablets) as well as AR-specific computingdevices, including wearable headsets and eyewear, for example.

The system includes the use of a controlled, real-world environment orspace. The given space is controlled, meaning the space itself andreal-world objects and articles, and other components within said space,are controlled, such as control over the appearance of walls, flooring,ceiling, placement of objects, lighting, temperature, and sounds, andthe like. In other words, many, if not all, aspects of the given spacemay be controlled to provide a specific environment in which to providean AR experience in that given space to users (i.e., guests, patrons,participants, or the like). By controlling the space, the system of thepresent invention is able to provide a persistently shared experiencededicated to the specific space.

For any given controlled space, a shared point is initially established(also referred to herein as “world origin point” or “world origin”).Establishing a world origin point within the controlled space allows forthe AR platform to place digital content relative to the world originpoint for subsequent rendering across multiple AR-capable devices. Thecontrolled space is digitally mapped, such that digital data associatedwith the controlled space, including the world-origin point coordinatedata, is stored for subsequent retrieval and use during rendering of ARcontent.

In addition, the system further relies on image tracking for alignmentpurposes. For example, the physical space can be decorated using imagemarker technology. Use of image marker technology allows for canonicallyestablished images to represent coordinates associated with the worldorigin point. For example, at the start of a given AR session orexperience, devices with image tracking technology can utilize one ormore image trackers (i.e. physical markers) within a given space tolocalize into the space and align the AR session to the world originpoint. The localized coordinates of each image marker along with aunique image marker identifier is stored for each image for subsequentretrieval and use by each device, thereby allowing devices to understandthe space without requiring any individual device setup.

The AR platform further coordinates the world origin point of a givencontrolled space with anchor-based localization to thereby align themultiple devices. In particular, each device may be running ananchor-based software algorithm unique to that device's given platform.Each participating AR-capable device (i.e., AR-headset, smartphone,tablet, or other computing device that is AR-capable) within thecontrolled space essentially agrees upon the established world originpoint, thereby allowing for digital content (e.g., images) toconsistently appear in the same, real world location in the controlledspace for each individual device as a result of one or more localizationand subsequent re-localization processes for each device, as describedin greater detail herein.

Upon a set of devices localizing into the controlled space using atleast one of the image tracking and cloud anchoring techniques, the ARplatform allows for dynamic, real-time localization across all devicesin the given space. Each device will determine, through a series ofchecks, whether to start generating temporary cloud anchors for moreaccurately sharing an AR experience with new devices that enter thespace. As image tracking can require positioning devices in closeproximity to image markers, temporary cloud anchors provide an advantageof allowing more devices to arbitrarily localize into the space withouthaving a multitude of viewers try to crowd into the same vantage point.

The system of the present invention further accounts for drift byproviding for automatic and repeated localization (i.e.,re-localization) for any device. One or more locations within a givencontrolled space may be designated as re-localization points, in whichany given user's proximity may be detected via a proximity sensor, suchas a near-field communication-based device. For example, proximitysensors may include Bluetooth Low-Energy (BLE) sensors. Upon beingdetected, a near-field communication-based sensor may communicate withthe AR platform and/or device and subsequently initiate are-localization process, in which the device will automatically attemptto re-localize (requiring no direct input or interaction from the user).Accordingly, the system of the present invention provides for continuousre-alignment of the dynamic world origin point through a combination ofthe use of the physical image markers as well as disparate cloudservices of each device to maintain the associated coordinatesconsistently across device software systems throughout the duration ofeach AR session/experience.

Accordingly, the system of the present invention addresses the drawbacksof current augmented reality systems by recognizing the potential of howexperiential augmented reality can be when experiencing such contenttogether by many at the same time. The AR platform provides forsynchronized sharing of AR content in real time and across multipleAR-capable devices, thereby allowing multiple users to experience thesame AR content rendering in real time and within a live, physicalenvironment or space, wherein such rendering of AR content is adapted toeach user's point of view. The synchronization of content allows formultiple users within the given space to more naturally interface withthe shared AR content as well as observe an identical combination ofdigital and physical reality, thereby simultaneously experiencing andinteracting with augmented reality environments. The AR platform allowsfor the display of AR content within the same physical location andorientation across multiple AR-capable devices, regardless of thedevices being from identical or different manufactures. By combiningdifferent device types together, the system of the present invention isaccessible by most device owners, providing similar AR experiences toboth the handheld mobile market (i.e., smartphones or tablets) and themore expensive lightweight eyewear market. Additionally, by integratingand leveraging multiple technologies (i.e., image tracking technology,cloud-based anchor systems, local persistent anchoring systems, andre-localization proximity sensors), the system of the present inventionis able to ensure constant re-localization that does not depend solelyon a single technology. Based on the communication capabilities (e.g.,network communications), reliability can be shared across the differentplatforms, thereby improving the overall AR experience for all users.

For the sake of clarity and ease of description, the systems describedherein and AR experiences provided by such systems may be implemented inan indoor environment, such as within a room or multiple rooms within abuilding or enclosed space, such as an indoor attraction. Morespecifically, the following embodiments describe the use of multiplecontrolled spaces that are part of an overall AR experience to beprovided to the users (i.e., multiple rooms or spaces at a particularvenue, such as a multiple spaces representing multiple exhibits at anAR-based zoo). However, it should be noted that systems of the presentinvention may be used to provide AR experiences in outdoor environments(i.e., such as military training or outdoor entertainment venues andattractions).

FIG. 1 illustrates one embodiment of an exemplary system 10 consistentwith the present disclosure. As shown, system 10 includes an augmentedreality (AR) platform 12. The AR platform 12 may be embodied on aninternet-based computing system/service. For example, the AR platform 12may be embodied on a cloud-based service, for example. The AR platform12 is configured to communicate and share data with one or more users15(a)-15(n) via computing devices 16(a)-16(n) over a network 18, forexample. The system 10 further includes one or more remote serversystems 14, which may be associated with one or more backend platformsor systems for one or more of the computing devices 16. For example, aswill be described in greater detail herein, each of the computingdevices may run platform-specific anchor-based localization processes,including, but not limited to, cloud anchoring processes, such asApple's ARKit, Google's ARCore, or Microsoft's Hololens & Azure systems.Accordingly, the remote server systems 14 may be associated with suchplatform-specific anchor-based localization processes.

In the present context, depending on the specific AR experience to beprovided and the particular use of the system, the users may includeguests, patrons, participants, students, or the like. For example, inone example, the system of the present invention may be particularlyuseful in the entertainment industry in which a given venue providesentertainment to multiple guests or patrons at once, such as a zoo,theme park, sporting event, or the like. Similarly, the systems of thepresent invention may be useful for educational purposes (i.e.,classroom environment in which the instructor and associated courselesson is provided to multiple students via an AR experience provided oneach student's AR-capable device) or military and/or law enforcementexercises (i.e., soldiers, military personnel, police officers, etc.)can train via customized training scenarios provided via an ARexperience, including multi-user combat situations).

The network 18 may represent, for example, a private or non-privatelocal area network (LAN), personal area network (PAN), storage areanetwork (SAN), backbone network, global area network (GAN), wide areanetwork (WAN), or collection of any such computer networks such as anintranet, extranet or the Internet (i.e., a global system ofinterconnected network upon which various applications or service runincluding, for example, the World Wide Web). In alternative embodiments,the communication path between the computing devices 16, and/or betweenthe computing devices 16 and AR platform 12, and/or between thecomputing devices 16 and remote server system(s) 14, and/or between theAR platform 12 and remote server system(s) 14, may be, in whole or inpart, a wired connection.

The network 18 may be any network that carries data. Non-limitingexamples of suitable networks that may be used as network 18 includeWi-Fi wireless data communication technology, the internet, privatenetworks, virtual private networks (VPN), public switch telephonenetworks (PSTN), integrated services digital networks (ISDN), digitalsubscriber link networks (DSL), various second generation (2G), thirdgeneration (3G), fourth generation (4G), fifth-generation (5G)cellular-based data communication technologies, Bluetooth radio, NearField Communication (NFC), the most recently published versions of IEEE802.11 transmission protocol standards, other networks capable ofcarrying data, and combinations thereof. In some embodiments, network 18is chosen from the internet, at least one wireless network, at least onecellular telephone network, and combinations thereof. As such, thenetwork 18 may include any number of additional devices, such asadditional computers, routers, and switches, to facilitatecommunications. In some embodiments, the network 18 may be or include asingle network, and in other embodiments the network 18 may be orinclude a collection of networks.

The AR platform 12 is configured to communicate and share data with thecomputing devices 16 associated with one or more users 15 as well as theremote server system(s). Accordingly, the computing device 16 may beembodied as any type of device for communicating with the AR platform 12and remote server system(s) 14, and/or other user devices over thenetwork 18. For example, at least one of the user devices may beembodied as, without limitation, any form of computing device capable ofrendering the intended AR experience provided, in part, via the ARplatform 12, such as a smartphone or tablet, which include camerahardware and associated display for providing a view of the real-worldenvironment (via a viewfinder on the display when a camera is capturinga live view of the real-world environment) and further rendering digitalcontent provided by the AR platform 12 overlaying the real-worldenvironment. In addition to the use of smartphones and/or tablets, theuser devices 16 may include AR-capable wearable headsets, such as, forexample, Microsoft® Hololens®, or other augmented reality and/or mixedreality headsets.

The AR platform 12 includes a mesh network of technologies integratedwith one another and used to ultimately establish alignment of digitalAR content, including rendering thereof, against the controlled physicalenvironment or space. The AR platform 12 ultimately allows for multipleusers to experience the same AR content rendering in real time, whereinsuch rendering of AR content is adapted to each user's point of viewwithin the controlled, real-world space, as will be described in greaterdetail herein.

It should be noted that embodiments of the system 10 of the presentdisclosure include computer systems, computer operated methods, computerproducts, systems including computer-readable memory, systems includinga processor and a tangible, non-transitory memory configured tocommunicate with the processor, the tangible, non-transitory memoryhaving stored instructions that, in response to execution by theprocessor, cause the system to perform steps in accordance with thedisclosed principles, systems including non-transitory computer-readablestorage medium configured to store instructions that when executed causea processor to follow a process in accordance with the disclosedprinciples, etc.

FIG. 2 is a block diagram illustrating the augmented reality (AR)platform 12 in greater detail. As shown, the AR platform 12 may includean interface 20, a data collection and management module 22, alocalization/re-localization module 24, an AR content creation,management, and distribution module 26, and various databases 28 forstorage of data. As will be described in greater detail herein, the ARplatform 12 is configured to communicate and share data with one or moreusers 15(a)-15(n) via computing devices 16(a)-16(n) over a network 18,for example.

FIG. 3 is a block diagram illustrating the various databases in greaterdetail. In particular, the various databases for storage of datainclude, but are not limited to, a user database 30 for storing profilesof users and their associated devices, for example, a physical spacedatabase 32 for storing data associated with controlled physical spacesfor one or more associated AR experiences, an image marker database 34for storing image marker data associated with one or more controlledphysical spaces, an anchor database 36 for storing anchor data of agiven device 16 during an AR experience, a localization/re-localizationdatabase 38 for storing localization (and re-localization) data of agiven device 16 during an AR experience, and an AR content database 40for storing AR content (i.e., digital images or other media) to betransmitted to the devices 16 as part of an AR experience of a givencontrolled space, such as images including one or more objects, composedby the AR platform 12 or provided thereto from an external source, to bedisplayed as overlays on views of the controlled, real-world space viathe device 16. The data collection and management module 22 may beconfigured to communicate and exchange data with each of the databases,as well as the other modules provided.

The interface 20 may generally allow a user to gain access to one ormore features of the AR services, which may include an interactiveinterface in which users may select certain inputs may adjust, orotherwise result in interaction with, a given AR experience. Theinterface 20 may also provide general information regarding the ARexperience (i.e., guidance in the form of a map or layout providingdirections to the next exhibit or previous exhibit, requests promptingthe user to take certain actions, such as actively initiating alocalization process, alerts indicating to the user that certain ARexperiences are available and or ready, etc.).

FIG. 4 is a block diagram illustrating at least one embodiment of acomputing device (i.e., smartphone or tablet) 16 a for communicatingwith the AR platform 12 and remote server system(s) 14 and forsubsequently conveying an AR experience to an associated user 15 basedon communication with at least the AR platform 12. The mobile device 16generally includes a computing system 100. As shown, the computingsystem 100 includes one or more processors, such as processor 102.Processor 102 is operably connected to communication infrastructure 304(e.g., a communications bus, cross-over bar, or network). The processor102 may be embodied as any type of processor capable of performing thefunctions described herein. For example, the processor may be embodiedas a single or multi-core processor(s), digital signal processor,microcontroller, or other processor or processing/controlling circuit.

The computing system 100 further includes a display interface 106 thatforwards graphics, text, sounds, and other data from communicationinfrastructure 104 (or from a frame buffer not shown) for display ondisplay unit 108. The computing system further includes input devices110. The input devices 110 may include one or more devices forinteracting with the mobile device 16, such as a keypad, microphone,camera, as well as other input components, including motion sensors, andthe like. For example, the mobile device 16 may include any variety ofsensors for capturing data related to at least one of a location of theuser within the controlled, physical space, a point of gaze of the userwithin the given space, a field of view of the user within the givenspace, as well as a physical setting and objects within the given space.The sensors may include one or more of a camera, motion sensor, andglobal positioning satellite (GPS) sensor. The motion sensor may beembodied as any type of sensor configured to capture motion data andproduce sensory signals. For example, the motion sensor may beconfigured to capture data corresponding to the movement of the deviceor lack thereof. The motion sensor may include, for example, anaccelerometer, an altimeter, one or more gyroscopes, or other motion ormovement sensor to produce sensory signals corresponding to motion ormovement of the device 16 and/or a magnetometer to produce sensorysignals from which direction of travel or orientation can be determined.The one or more motion sensors may further include, or be coupled to, aninertial measurement unit (IMU) module for example.

The motion sensors may also be embodied as a combination of sensors,each of which is configured to capture a specific characteristic of themotion of the device 16, or a specific characteristic of user movement.A motion sensor embodied as a combination of sensors may use algorithms,such as, for example, fusion algorithms, to correct and compensate thedata from individual sensors and provide more robust motion sensing anddetection context than each individual sensor can provide alone.

In one embodiment, the display unit 108 may include a touch-sensitivedisplay (also known as “touch screens” or “touchscreens”), in additionto, or as an alternative to, physical push-button keyboard or the like.The touch screen may generally display graphics and text, as well asprovides a user interface (e.g., but not limited to graphical userinterface (GUI)) through which a user may interact with the mobiledevice 16, such as accessing and interacting with applications executedon the device 16, including an app for communicating and exchanging datawith the AR platform 12, as well as rendering digital AR contentprovided by the AR platform 12.

The computing system 100 further includes main memory 112, such asrandom access memory (RAM), and may also include secondary memory 114.The main memory 112 and secondary memory 114 may be embodied as any typeof device or devices configured for short-term or long-term storage ofdata such as, for example, memory devices and circuits, memory cards,hard disk drives, solid-state drives, or other data storage devices.Similarly, the memory 112, 114 may be embodied as any type of volatileor non-volatile memory or data storage capable of performing thefunctions described herein.

In the illustrative embodiment, the mobile device 16 may maintain one ormore application programs, databases, media and/or other information inthe main and/or secondary memory 112, 114. The secondary memory 114 mayinclude, for example, a hard disk drive 116 and/or removable storagedrive 118, representing a floppy disk drive, a magnetic tape drive, anoptical disk drive, etc. Removable storage drive 118 reads from and/orwrites to removable storage unit 120 in any known manner. The removablestorage unit 120 may represent a floppy disk, magnetic tape, opticaldisk, etc. which is read by and written to by removable storage drive118. As will be appreciated, removable storage unit 120 includes acomputer usable storage medium having stored therein computer softwareand/or data.

In alternative embodiments, the secondary memory 114 may include othersimilar devices for allowing computer programs or other instructions tobe loaded into the computing system 100. Such devices may include, forexample, a removable storage unit 124 and interface 122. Examples ofsuch may include a program cartridge and cartridge interface (such asthat found in video game devices), a removable memory chip (such as anerasable programmable read only memory (EPROM), or programmable readonly memory (PROM)) and associated socket, and other removable storageunits 124 and interfaces 122, which allow software and data to betransferred from removable storage unit 124 to the computing system 100.

The computing system 100 further includes one or more applicationprograms 126 directly stored thereon. The application program(s) 126 mayinclude any number of different software application programs, eachconfigured to execute a specific task.

The computing system 100 further includes a communications interface128. The communications interface 128 may be embodied as anycommunication circuit, device, or collection thereof, capable ofenabling communications between the mobile device 16 external devices(other mobile devices 16, the AR platform 12 and/or remote serversystem(s) 14). The communications interface 128 may be configured to useany one or more communication technology and associated protocols, asdescribed above, to effect such communication. For example, thecommunications interface 128 may be configured to communicate andexchange data with the digital content management platform 12, and/orone other mobile device 16, via a wireless transmission protocolincluding, but not limited to, Bluetooth communication, infraredcommunication, near field communication (NFC), radio-frequencyidentification (RFID) communication, cellular network communication, themost recently published versions of IEEE 802.11 transmission protocolstandards, and a combination thereof. Examples of communicationsinterface 128 may include a modem, a network interface (such as anEthernet card), a communications port, a Personal Computer Memory CardInternational Association (PCMCIA) slot and card, wireless communicationcircuitry, etc.

Computer programs (also referred to as computer control logic) may bestored in main memory 112 and/or secondary memory 114 or a localdatabase on the mobile device 16. Computer programs may also be receivedvia communications interface 128. Such computer programs, when executed,enable the computing system 100 to perform the features of the presentinvention, as discussed herein. In particular, the computer programs,including application programs 126, when executed, enable processor 102to perform the features of the present invention. Accordingly, suchcomputer programs represent controllers of computer system 100.

In one embodiment where the invention is implemented using software, thesoftware may be stored in a computer program product and loaded into thecomputing system 100 using removable storage drive 118, hard drive 116or communications interface 128. The control logic (software), whenexecuted by processor 102, causes processor 102 to perform the functionsof the invention as described herein.

In another embodiment, the invention is implemented primarily inhardware using, for example, hardware components such as applicationspecific integrated circuits (ASICs). Implementation of the hardwarestate machine so as to perform the functions described herein will beapparent to persons skilled in the relevant art(s).

In yet another embodiment, the invention is implemented using acombination of both hardware and software.

FIG. 5 is a block diagram illustrating at least one embodiment of acomputing device (i.e., wearable headset) 16 b for communicating withthe AR platform 12 and for subsequently conveying an AR experience to anassociated user 15 based on communication with at least the AR platform12. The headset 16 b includes a display unit 200 positioned to be withina field of view of a person wearing the headset (i.e., the “wearer”) anda processing subsystem 202 built into the headset 16 b and configured tocommunicate with the AR platform 12 and remote server system(s) 14 toexchange various sensor data to be used for at least one oflocalization, re-localization, and eventual receipt of augmented reality(AR) content to be displayed on the display unit 100. The processingsubsystem 202 includes, for example, a hardware processor coupled tonon-transitory, computer-readable memory containing instructionsexecutable by the processor to cause the processing subsystem 200 tocommunicate with the AR platform 12 and remote server system(s) 14 overthe network 18 and exchange data therewith.

As shown, the headset 16 b may include a variety of sensors 204 forcapturing data related to at least one of a location of the wearerwithin the controlled, physical space, a point of gaze of the wearerwithin the physical space, a field of view of the wearer within thephysical space, and a physical setting and objects within the space. Thesensors 204 may include one or more of a camera 206, motion sensor 208,and global positioning satellite (GPS) sensor 210.

The camera 206 is operable to capture one or more images (or a series ofimages) of the given, controlled space in which the AR experience istaking place. The motion sensor 208 may include an accelerometer, analtimeter, one or more gyroscopes, other motion or movement sensors toproduce sensory signals corresponding to motion or movement of theheadset 16 b and the wearer, and a magnetometer to produce sensorysignals from which direction of travel or orientation of the headset 16b (i.e., the orientation of the wearer) can be determined.

The motion sensor 208, for example, may be embodied as any type ofsensor configured to capture motion data and produce sensory signals.For example, the motion sensor may be configured to capture datacorresponding to the movement of the device or lack thereof. The motionsensor may include, for example, an accelerometer, an altimeter, one ormore gyroscopes, or other motion or movement sensor to produce sensorysignals corresponding to motion or movement of the headset 16 b and/or amagnetometer to produce sensory signals from which direction of travelor orientation can be determined. The one or more motion sensors mayfurther include, or be coupled to, an inertial measurement unit (IMU)module for example.

The motion sensors may also be embodied as a combination of sensors,each of which is configured to capture a specific characteristic of themotion of the headset 16 b, or a specific characteristic of usermovement. A motion sensor embodied as a combination of sensors may usealgorithms, such as, for example, fusion algorithms, to correct andcompensate the data from individual sensors and provide more robustmotion sensing and detection context than each individual sensor canprovide alone.

FIG. 6 shows a perspective view of an exemplary wearable headset 16 b ofthe system of the present invention. As illustrated, the headset 16 b isgenerally in form of a pair of eyewear. The headset 16 b includes aframe member 216 including a right earpiece 218 and a left earpiece 220,which may be fixedly or hingedly attached to the frame member 216. Theframe member 216 further includes a center bridge 222. The headset 16 bincludes a first lens 224 (e.g., as a right lens) and also includes asecond lens 226 (e.g., as a left lens) to provide binocular vision. Theright lens 224 and left lens 226 are mounted to the frame member 216.The headset 16 b may be dimensioned to be worn on a human head, witheach earpiece extending over a respective ear such that a portion of theframe member 216 extends across the human face. The right lens 224 andleft lens 226 may be mounted to the frame member 216 such that, when theheadset 16 b is worn, each of the right lens and left lens 224, 226 isdisposed in front of a the respective eyes of the wearer. As previouslydescribed, the headset 16 b may include one or more sensors 232, 234,236, and 238, such as camera(s), microphone(s), motion sensor(s), GPSsensor(s), and the like, for capturing/sensing data associated with thelocation, orientation, or field-of-view information of the personwearing the headset 16 b to compose the augmented reality content inreal-time. Furthermore, in certain embodiments, the headset 16 bincludes one or more of electronic displays or projectors 228, 230 foreach of the right lens and left lens 224, 226, as previously describedherein.

FIG. 7 is a block diagram illustrating communication between multipleAR-capable devices 16 a and 16 b and the AR platform 12 for localizationand re-localization thereof based on at least one of image trackingtechnology, anchor-based technology, and proximity sensing.

As previously described, the system 12 includes the use of a physical,real-world environment, preferably a controlled space (i.e., a room orat least partially enclosed space) in which the AR content is to bepresented to the multiple users (via each user's AR-capable device). Theuse of a controlled space allows for the system of the present inventionto provide a persistently shared experience dedicated to the specificspace. In some embodiments, the environment may include multiplecontrolled spaces that are part of an overall AR experience to beprovided to the users (i.e., multiple rooms or spaces at a particularvenue, such as a multiple spaces representing multiple exhibits at anAR-based zoo).

For any given controlled space, a shared point is initially established(also referred to herein as “world origin point” or “world origin”). Theworld origin point is generally defined as a specific position andorientation within the given space, which may be based on coordinatedata (e.g., a coordinate axis system, including an x,y,z position andx,y,z orientation). Once established, all digital content will besubsequently placed relative to that world origin point. In layman'sterms, the world origin point on a canonical world map would be thelatitude and longitude of (0,0) with an orientation of north pointing tothe north pole. All location coordinates specified with latitude andlongitude values can be reasonably understood by any map program thatrespects this world origin point, with the latitude and longitudecoordinates considered as being relative to that known world originpoint.

Establishing a world origin point within the controlled space allows forthe AR platform 12 to place digital content relative to the world originpoint for subsequent rendering across multiple AR-capable devices. Thecontrolled space is digitally mapped, such that digital data associatedwith the controlled space, including the world-origin point coordinatedata, is stored within the physical space database 32, for example, forsubsequent retrieval and use during rendering of AR content.

The system 10 further relies on image tracking for alignment purposes.For example, the physical space can be decorated using image markertechnology. Use of image marker technology allows for canonicallyestablished images to represent coordinates associated with the worldorigin point. For example, at the start of a given AR session orexperience, devices with image tracking technology can utilize one ormore image trackers (i.e. physical markers) within a given space tolocalize into the space and align the AR session to the world originpoint. The localized coordinates of each image marker along with aunique image marker identifier data is stored for each image within theimage marked database 34, for example, for subsequent retrieval and useby each device 16, thereby allowing devices to understand the spacewithout requiring any individual device setup.

The AR platform 12 further coordinates the world origin point of a givencontrolled space with anchor-based localization to thereby align themultiple devices. In particular, each device 16 may be running ananchor-based software algorithm unique to that device's given platform.Anchors are understood to include generated locations that represent aphysical location of the associated device in the real world and storedas serialized data (e.g., in the form of coordinate data), and may bestored within the anchor database 36, for example. In some embodiments,the devices 16 may be running respective cloud anchoring systems.Additionally, some devices 16 may be running respective persistentanchoring systems. Accordingly, each of the devices 16 may runplatform-specific anchor-based localization processes, including, butnot limited to, cloud anchoring processes, such as Apple's ARKit,Google's ARCore, or Microsoft's Hololens & Azure systems.

As an anchor represents a physical point in the real world, anchors uselocalization to identify their relative location to world origincoordinates for each individual AR session, and thus those coordinateswill vary with each session while their location and orientation wouldbe identical across sessions (with a small margin of error depending onplatform accuracy). Each participating device 16 within the controlledspace essentially agrees upon the established world origin point,thereby allowing for digital content (e.g., images) to consistentlyappear in the same, real world location in the controlled space for eachindividual device as a result of one or more localization and subsequentre-localization processes for each device 16.

For each cloud anchoring system, for example, anchors will beestablished for each integrated platform in a similar manner to imagemarkers. However, in the present system, cloud anchors are establishedusing a computer vision-based mesh understanding of the physical world.As previously described, each device within the controlled spaceessentially agrees upon the established world origin point, such thateach device localizes into the space based, at least in part, onestablished anchors for that device (i.e., correlation of anchor datawith world origin point data).

Upon the devices 16 localizing into the controlled space using at leastone of the image tracking and cloud anchoring techniques, the ARplatform 12 allows for dynamic, real-time localization across alldevices in the given space, as carried out via thelocalization/re-localization module 24 in some instances. In someembodiments, each device 16 will determine, through a series of checks,whether to start generating temporary cloud anchors for more accuratelysharing an AR experience with new devices that enter the space. As imagetracking can require positioning devices in close proximity to imagemarkers, temporary cloud anchors provide an advantage of allowing moredevices to arbitrarily localize into the space without having amultitude of viewers try to crowd into the same vantage point.

The system 10 further accounts for drift. For example, devices may becontinuously re-localizing into the real world through a series ofsensors, which may include an RGB camera, Lidar sensors, inertialmeasurement unit (IMU), motion sensors, infrared, or other trackingsystem. Such sensors are all subject to disruption, which can interferewith the device's understanding of its position and orientation in thereal world environment. Accordingly, as a result of such disruption, thedigital AR content provided may shift from its originally localizedworld origin, resulting in a phenomenon known as drift, which can causedigitally placed objects to shift to incorrect locations as a result.

To counter the effects of drift and to make the system easy to use foreach user, the system of the present invention provides for automaticand repeated localization (i.e., re-localization) for any device. Inparticular, for a given AR experience that may include multiplecontrolled spaces (e.g., multiple exhibits in an AR-based zoo, forexample), multiple locations within the real world environment may bedesignated as re-localization points, in which any given user'sproximity may be detected via a proximity sensor, such as a near-fieldcommunication-based device. For example, proximity sensors may includeBluetooth Low-Energy (BLE) sensors 13. Upon being detected, a near-fieldcommunication-based sensor may communicate with the AR platform 12and/or device 16 and subsequently initiate a re-localization process, inwhich the device 16 will automatically attempt to re-localize (requiringno direct input or interaction from the user), wherein re-localizationdata can be stored within the localization/re-localization database 38.Such re-localization points can be placed throughout a given ARexperience at regular intervals that users (i.e., guests orparticipants) must necessarily pass through and are encouraged to comecloser as part of the attraction(s). Accordingly, the system 10 of thepresent invention provides for continuous re-alignment of the dynamicworld origin point through a combination of the use of the physicalimage markers as well as disparate cloud services of each device tomaintain the associated coordinates consistently across device softwaresystems throughout the duration of each AR session/experience.

As previously described, each device 16 transmits data, including sensordata and images or other information related to the user, to the ARplatform 12. In turn, the AR platform 12 processes the data (via the ARcontent creation, management, and distribution module 26) in accordancewith AR-based processes and in accordance with AR software, such asAutoCad3D, StudioMax or Cinema4D programs. The AR processing may berecognition-based augmented reality or location-based augmented reality,or a combination of both, as generally understood. The AR platform 12may then obtain and/or create AR content, which may be in the form ofone or more images including one or more objects, to be displayed asoverlays on views of the physical, real-world space. In particular,platform 12 may use the location, orientation, or field-of-viewinformation of the user, as well as other data associated with thedevice 16 (image marked data, anchor data, localization(re-localization) data, etc.) to compose the AR content in real, ornear-real, time. Accordingly, the sensor data is important and is reliedupon by the platform 12, which is able to generate and reposition ARcontent according to a location of the user (and associated device)within the physical space, as well as a position of the wearer's headwith regards to objects within the given space. The devices effectivelyimmerse the user in the augmented reality experience, because elementsof the augmented reality scene are updated and received on-the-fly.

FIG. 8 is an exemplary layout of a venue comprised of multiplecontrolled spaces, each having space an established world origin pointfor a given attraction, at least one image tracking marker, andre-localization zones. As shown, guests may enter a first space (Space1), in which each guest is equipped with an AR-capable device. Theirdevices will undergo an initial localization process, in which thedevice localizes to the map using at least one of the image tracking andcloud anchoring techniques previously described herein, depending onwhich technique is available. If both are available, the cloud anchoringlocalization takes priority. The next six spaces (Space 2 through Space6) consist of exhibits, each including a controlled physical spaceproviding a respective AR experience to multiple guests. Each of theexhibit spaces include re-localization areas or zones. As previouslydescribed, such zones may generally be implemented as re-localizationpoints, in which any given user's proximity may be detected via aproximity sensor. Upon being detected, a re-localization process isinitiated behind the scenes (the user is not aware). The docent may givean introduction to the area (space), thereby providing a bit more timefor the re-localization process to complete, before moving into the roomto receive the AR experience for that given space. It should be notedthat, as a fallback option, image tracking markers may be placedthroughout each space if needed. Further, as shown, re-localizationpoints can be placed throughout a given AR experience at regularintervals that the guests must necessarily pass through and areencouraged to come closer as part of the attraction(s). Accordingly, thesystem 10 of the present invention provides for continuous re-alignmentof the dynamic world origin point through a combination of the use ofthe physical image markers as well as disparate cloud services of eachdevice to maintain the associated coordinates consistently across devicesoftware systems throughout the duration of each AR session/experience.

The venue may further include a couple final spaces in which the guestsunload and remove the devices (space 7) once the AR experience iscomplete (once the guest has passed through all exhibits) and the guestcan further and enter the gift shop (space 8) to purchase items or leavethe venue.

FIGS. 9A-9F show a continuous flow diagram illustrating a method 300 forinitial localization of one or more AR-capable devices within acontrolled environment or space prior to commencing an AR experience orsession. The method includes starting up the device (i.e., turning onthe power) within the given space in which the AR experience will takeplace. Upon starting up the device, the user must wait for localizationprocesses to begin within the given environment or space (operation302), in which the device will be communicating with at least one of theAR platform 12 and remote server system(s) 14, exchanging datatherewith. Upon localizing with the physical environment (operation304), a determination is then made in operation 306 as to whether thereis any saved spatial anchor data available or present. At this point,the databases 28 are analyzed to determine if any saved spatial anchordata is available. If it is determined in operation 306 that spatialanchor data is available/present, then a first anchor (presumably afirst anchor tied or associated with the saved spatial anchor data) isloaded (operation 308). A determination is then made in operation 312 asto whether the first anchor is local or a cloud anchor.

If it is determined that the first anchor is local, then a determinationis made in operation 314 as to whether the local anchor is able to belocalized. If it is determined that the local anchor is able to belocalized, then an AR experience is localized (operation 316) and thedevice is then connected to the multiplayer/multi-user network(operation 320). If it is determined that the local anchor is unable tobe localized, then a determination is made in operation 322 as towhether there are additional anchors (presumably tied to or associatedwith the saved spatial anchor data) to check. If it is determined thatthere are no additional anchors to check, then image localization(utilizing image tracking technology described herein) is attempted(operation 310). If it is determined that there are additional anchorsto check, then the determination in operation 312 (as to whether thefirst anchor is local or a cloud anchor) is repeated.

If it is determined (in operation 312) that the first anchor iscloud-based, then a determination is made in operation 316 as to whetherit is possible to localize the cloud-based anchor with the associatedcloud-based server. If it is determined that the cloud-based anchor isable to be localized with the cloud, then an AR experience is localized(operation 316) and the device is then connected to themultiplayer/multi-user network (operation 320). If it is determined thatthe cloud-based anchor is unable to be localized with the cloud, then adetermination is made in operation 322 as to whether there areadditional anchors (presumably tied to or associated with the savedspatial anchor data) to check. If it is determined that there are noadditional anchors to check, then image localization (utilizing imagetracking technology described herein) is attempted (operation 310). Ifit is determined that there are additional anchors to check, then thedetermination in operation 312 (as to whether the first anchor is localor a cloud anchor) is repeated.

Reverting back to operation 306, if it is determined that there spatialanchor data is not available or present, then image localization isattempted (operation 310). Upon attempting image localization, thedevice enters an image scanning mode (operation 324). A determination isthen made in operation 326 as to whether any image tracking targets ormarkers are found/detected. If it is determined that image trackertargets or markers are found/detected, then then AR experience islocalized (operation 316) and the device is then connected to themultiplayer/multi-user network (operation 320). If it is determined thatimage tracker targets or markers are not found/detected, then temporarylocalization is created (operation 328) and the device is then connectedto the multiplayer/multi-user network (operation 320).

Upon connecting to the multiplayer/multi-user network, a determinationis then made in operation 330 as to whether the AR experience islocalized. If it is determined that the AR experience is localized, thena subsequent determination is made in operation 332 as to whether thereare any currently shared network anchors. If it is determined that thereare currently shared network anchors, then the AR experience is started(operation 334). If it is determined that there are no currently sharednetwork anchors, then networked anchors are created and shared(operation 336) and the AR experience is then started (operation 334).

If it is determined in operation 330 that the AR experience is notlocalized, then a determination is made in operation 338 as to whetherthere are any currently shared network anchors. If it is determined thatthere are currently shared network anchors available, then a firstanchor is loaded (operation 340) and a subsequent determination is madein operation 342 as to whether the anchor can be localized with a cloud.If it is determined that the first anchor can be localized with a cloud,then an AR experience is localized (operation 346) and the AR experienceis started (operation 334). If it is determined that the first anchor isunable to be localized with a cloud, then a determination is made inoperation 348 as to whether there are additional anchors to check. If itis determined that there are additional anchors to check, then thedetermination in operation 342 is repeated.

Reverting back to operation 338, if it is determined that there are nocurrently shared network anchors, then networked anchors are created andshared (operation 350), and then the AR experience is started (operation334).

FIG. 10 is a block diagram of one embodiment of a method for initiatinga re-localization session of an AR-enabled device. FIG. 11 is a blockdiagram of another embodiment of a method for initiating are-localization session of an AR-enabled device.

The method illustrated in FIG. 10 generally corresponds to a scenario inwhich a guest is at a venue comprised of multiple controlled spaces,such as an AR-based zoo with multiple exhibits. In the first scenario(see FIG. 10) a guest may generally enter a controlled space with theassociated AR-capable device. Upon entering the space, a proximitysensor (such as a BLE sensor) detects that the guest entered a new zone(see FIG. 8), and triggers a re-localization scanning event. The gueststops at an entryway while the docent introduces the room, encouragingguests to look around the room. As the guest is stopped at the entryway,the device scans the geometry data and attempts to find re-localizationanchors through local or cloud correlation/comparison. Uponre-localization, the device indicates to the guest that the device isnow “ready”. Once all of the devices from each of the guests within thatspace are “ready”, then the docent is notified to continue thetour/experience. Accordingly, upon receiving the “ready” notification,the guest then proceeds into the given space to continue the ARexperience.

The method illustrated in FIG. 11 generally corresponds to a scenario inwhich a guest is having difficulties with their AR experience, such aspoor feedback or visibility of AR content. Accordingly, the guest mayutilize an interface to select a help or assistance feature. In turn,the guest may be presented with a mini-map or layout of a specific zoneor space and they may be directed to the nearest image tracking markerassociated with that given zone or space so as to re-localize to thespecific scene based on image tracking localization techniques describedherein. In turn, the device turns on the re-localization mode and beingsscanning the given space for re-localization while the guest uses imagetracking to attempt to re-localize as well. Upon successfulre-localization, the device turns off the re-localization mode and theAR experience resumes.

By providing a truly immersive and shared AR experience, systems of thepresent invention can be particularly beneficial in various industriesthat cater to, or otherwise rely on, multiple guests, participants,patrons, or the like.

In the present context, depending on the specific AR experience to beprovided and the particular use of the system, the users may includeguests, patrons, participants, students, or the like. For example, inone example, the system of the present invention may be particularlyuseful in the entertainment industry in which a given venue providesentertainment to multiple guests or patrons at once, such as a zoo,theme park, sporting event, or the like. Similarly, the systems of thepresent invention may be useful for educational purposes (i.e.,classroom environment in which the instructor and associated courselesson is provided to multiple students via an AR experience provided oneach student's AR-capable device) or military and/or law enforcementexercises (i.e., soldiers, military personnel, police officers, etc.)can train via customized training scenarios provided via an ARexperience, including multi-user combat situations).

FIGS. 12-20 are images depicting various implementations and uses of thesystem of the present invention.

FIGS. 12-16 are images depicting an AR-based zoo experience for multipleguests within controlled spaces (i.e., specific “exhibits”), in whichsystems of the present invention provide synchronized sharing ofzoo-based AR content (i.e., zoo-related animals) in real time and acrossmultiple AR-capable devices (i.e., wearable headsets and/or personalcomputing devices, such as a smartphone or tablet).

FIG. 17 is an exemplary layout or map of an AR-based zoo experience,illustrating the various “exhibits”.

FIG. 18 is an image depicting an AR-based classroom experience, in whichthe instructor and associated course content is provided to multiplestudents via an AR experience.

FIG. 19 is an image depicting another embodiment of an AR-basedclassroom experience, in which the instructor and associated coursecontent/lesson is provided to multiple students via an AR experience,wherein each student is viewing and interacting with the coursecontent/lesson and instructor via a tablet computing device, furtherillustrating the multiple point of views for each student, adding to therealism and feel. Such an experience is particularly useful for distanceeducation, such as remote learning or the like.

FIG. 20 is an image depicting an AR-based military experience, in whichmultiple soldiers are provided with a military training scenario.

Accordingly, the system of the present invention addresses the drawbacksof current augmented reality systems by recognizing the potential of howexperiential augmented reality can be when experiencing such contenttogether by many at the same time. The AR platform provides forsynchronized sharing of AR content in real time and across multipleAR-capable devices, thereby allowing multiple users to experience thesame AR content rendering in real time and within a live, physicalenvironment or space, wherein such rendering of AR content is adapted toeach user's point of view. The synchronization of content allows formultiple users within the given space to more naturally interface withthe shared AR content as well as observe an identical combination ofdigital and physical reality, thereby simultaneously experiencing andinteracting with augmented reality environments. The AR platform allowsfor the display AR content within the same physical location andorientation across multiple AR-capable devices, regardless of thedevices being from identical or different manufactures. Furthermore, bycombining different device types together, the system of the presentinvention is accessible by most device owners, providing handheld mobileAR (i.e., by way of smartphones or tablets) to more expensivelightweight eyewear. Furthermore, by integrating and leveraging multipletechnologies (i.e., image tracking technology, cloud-based anchorsystems, local persistent anchoring systems, and re-localizationproximity sensors), the system of the present invention is able toensure constant re-localization that does not depend solely on a singletechnology. Based on the communication capabilities, via the ARplatform, reliability can be shared across the different platforms,thereby improving the overall AR experience by countering drift.

As used in any embodiment herein, the term “module” may refer tosoftware, firmware and/or circuitry configured to perform any of theaforementioned operations. Software may be embodied as a softwarepackage, code, instructions, instruction sets and/or data recorded onnon-transitory computer readable storage medium. Firmware may beembodied as code, instructions or instruction sets and/or data that arehard-coded (e.g., nonvolatile) in memory devices. “Circuitry”, as usedin any embodiment herein, may comprise, for example, singly or in anycombination, hardwired circuitry, programmable circuitry such ascomputer processors comprising one or more individual instructionprocessing cores, state machine circuitry, and/or firmware that storesinstructions executed by programmable circuitry. The modules may,collectively or individually, be embodied as circuitry that forms partof a larger system, for example, an integrated circuit (IC), systemon-chip (SoC), desktop computers, laptop computers, tablet computers,servers, smartphones, etc.

Any of the operations described herein may be implemented in a systemthat includes one or more storage mediums having stored thereon,individually or in combination, instructions that when executed by oneor more processors perform the methods. Here, the processor may include,for example, a server CPU, a mobile device CPU, and/or otherprogrammable circuitry.

Also, it is intended that operations described herein may be distributedacross a plurality of physical devices, such as processing structures atmore than one different physical location. The storage medium mayinclude any type of tangible medium, for example, any type of diskincluding hard disks, floppy disks, optical disks, compact diskread-only memories (CD-ROMs), compact disk rewritables (CD-RWs), andmagneto-optical disks, semiconductor devices such as read-only memories(ROMs), random access memories (RAMs) such as dynamic and static RAMs,erasable programmable read-only memories (EPROMs), electrically erasableprogrammable read-only memories (EEPROMs), flash memories, Solid StateDisks (SSDs), magnetic or optical cards, or any type of media suitablefor storing electronic instructions. Other embodiments may beimplemented as software modules executed by a programmable controldevice. The storage medium may be non-transitory.

As described herein, various embodiments may be implemented usinghardware elements, software elements, or any combination thereof.Examples of hardware elements may include processors, microprocessors,circuits, circuit elements (e.g., transistors, resistors, capacitors,inductors, and so forth), integrated circuits, application specificintegrated circuits (ASIC), programmable logic devices (PLD), digitalsignal processors (DSP), field programmable gate array (FPGA), logicgates, registers, semiconductor device, chips, microchips, chip sets,and so forth.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. Thus, appearances of the phrases “in oneembodiment” or “in an embodiment” in various places throughout thisspecification are not necessarily all referring to the same embodiment.Furthermore, the particular features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments.

The term “non-transitory” is to be understood to remove only propagatingtransitory signals per se from the claim scope and does not relinquishrights to all standard computer-readable media that are not onlypropagating transitory signals per se. Stated another way, the meaningof the term “non-transitory computer-readable medium” and“non-transitory computer-readable storage medium” should be construed toexclude only those types of transitory computer-readable media whichwere found in In Re Nuijten to fall outside the scope of patentablesubject matter under 35 U.S.C. § 101.

The terms and expressions which have been employed herein are used asterms of description and not of limitation, and there is no intention,in the use of such terms and expressions, of excluding any equivalentsof the features shown and described (or portions thereof), and it isrecognized that various modifications are possible within the scope ofthe claims. Accordingly, the claims are intended to cover all suchequivalents.

INCORPORATION BY REFERENCE

References and citations to other documents, such as patents, patentapplications, patent publications, journals, books, papers, webcontents, have been made throughout this disclosure. All such documentsare hereby incorporated herein by reference in their entirety for allpurposes.

EQUIVALENTS

Various modifications of the invention and many further embodimentsthereof, in addition to those shown and described herein, will becomeapparent to those skilled in the art from the full contents of thisdocument, including references to the scientific and patent literaturecited herein. The subject matter herein contains important information,exemplification and guidance that can be adapted to the practice of thisinvention in its various embodiments and equivalents thereof.

1. A system comprising: a plurality of augmented reality (AR)-capabledevices for providing respective users with an augmented view of areal-world environment; and an augmented reality (AR) platformconfigured to communicate and exchange data with each of the pluralityof AR-capable devices over a network, the AR platform comprising ahardware processor coupled to non-transitory, computer-readable memorycontaining instructions executable by the processor to cause the ARplatform to: receive, from each of the AR-capable devices, localizationdata associated with at least one of an anchor-based system and an imagetracking-based system for establishing a location of a respectiveAR-capable device within the real-world environment; process thelocalization data and assign a location of each AR-capable devicerelative to a shared, fixed origin point within the real-worldenvironment; and transmit AR content to each of the AR-capable devices,the AR content configured to be displayed and rendered by eachAR-capable device based, at least in part, on the assigned location ofeach respective AR-capable device.
 2. The system of claim 1, wherein theanchor-based system comprises a cloud anchor system.
 3. The system ofclaim 1, wherein the anchor-based system comprises a persistent anchorsystem.
 4. The system of claim 1, wherein processing of the localizationdata comprises synchronous alignment of the localization data from eachof the AR-capable devices relative to the shared, fixed origin pointwithin the real-world environment.
 5. The system of claim 4, wherein theAR content is configured to be displayed and rendered by each AR-capabledevice such that one or more digital images associated with the ARcontent appears at an identical location within the real-worldenvironment.
 6. The system of claim 1, wherein the AR-capable devicescomprise at least one of a smartphone, a tablet, and a wearable headsetor eyewear.
 7. The system of claim 6, wherein the AR platform isconfigured to receive additional data from each of the AR-capabledevices associated with at least one of, a point of gaze of theassociated user within the real-world environment, a field of view ofthe user within the real-world environment, and a physical setting andobjects within the real-world environment.
 8. The system of claim 7,wherein each of the AR-capable devices comprises one or more sensorsselected from the group consisting of a camera, a motion sensor, and aglobal positioning satellite (GPS) sensor.
 9. The system of claim 8,wherein the motion sensor is selected from the group consisting of anaccelerometer, an altimeter, one or more gyroscopes, other motion ormovement sensors to produce sensory signals corresponding to motion ormovement of the AR-capable device, and a magnetometer to produce sensorysignals from which direction of travel or orientation of the AR-capabledevice can be determined.
 10. The system of claim 6, wherein eachAR-capable device comprises a display unit for providing respectiveusers with an augmented view of the real-world environment
 11. Thesystem of claim 10, wherein the display unit comprises a lens and adigital display.
 12. The system of claim 11, wherein, when wearing theheadset or eyewear, a user can view the real-world environment throughat least a portion of the lens and view AR content projected via thedigital display as objects in the real-world environment.
 13. The deviceof claim 10, wherein said display unit comprises a digital displayselected from the group consisting of a light-emitting diode (LED)projector and/or display, an organic light-emitting diode (OLED)projector and/or display, a liquid crystal display (LCD), a liquidcrystal on silicon (LCOS) projector and/or display, and a microdisplayand/or microprojector.
 14. The system of claim 1, wherein the AR contentcomprises one or more objects rendered as an overlay on the real-worldenvironment.
 15. The system of claim 14, wherein the AR contentcomprises a game.
 16. The system of claim 14, wherein the AR contentcomprises an interactive game experience.
 17. The system of claim 14,wherein the AR content is selected from the group consisting of avirtual object, a digital image, a digital video, an application, ascript, a promotion, an advertisement, a graphic, and an animation. 18.The system of claim 1, wherein the real-world environment is associatedwith an area of interest or attraction.
 19. The system of claim 18,wherein the real-world environment is associated with a zoo.
 20. Thesystem of claim 18, wherein the real-world environment is associatedwith a stadium or other setting for sporting events, concerts, or otherentertainment-related events.